The present invention relates generally to the field of determining the occupancy state of the vehicle which entails sensing, detecting, monitoring and/or identifying various objects, and parts thereof, which are located within the passenger compartment of the vehicle. The occupancy state is a broad or narrow description of the state or condition of one or more occupying items in the vehicle. Thus, a determination of the occupancy state may include a determination of the type or class of any occupying items, the size of any occupying items, the position of any occupying item including the orientation of occupying items, the identification of any occupying items and/or the status of any occupying items (whether the occupying items are conscious or unconscious). The determination of the occupancy state is used to control a vehicular component.
In particular, the present invention relates to an efficient and highly reliable system for evaluating the occupancy of a vehicle by detecting the presence and optionally orientation of objects in the seats of the passenger compartment, e.g., a rear facing child seat (RFCS) situated in the passenger compartment in a location where it may interact with a deploying occupant protection apparatus, such as an airbag, and/or for detecting an out-of-position occupant. The system permits the control and selective suppression of deployment of the occupant protection apparatus when the deployment may result in greater injury to the occupant than the crash forces themselves. This is accomplished in part through a specific placement of transducers of the system, the use of a pattern recognition system, possibly a trained neural network and combinations of neural networks called modular neural, voting or ensemble neural networks, and/or a novel analysis of the signals from the transducers.
1. Prior Art on Sensing of Out-of-position Occupants and Rear Facing Child Seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and thus significant improvements to the airbag system are necessary. As discussed in detail in one or more of the patents and patent applications cross-referenced above, for a variety of reasons, vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of any deployment thereof. Also, a child in a rear facing child seat which is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. xe2x80x9cHow Airbags Workxe2x80x9d presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection is required in order to minimize the damages caused by deploying airbags. It also may be required in order to minimize the damage caused by the deployment of other types of occupant protection and/or restraint devices which might be installed in the vehicle.
Initially, these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded and unwanted airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system is therefore needed to detect the presence of occupants, determine if they are out-of-position, defined below, and to identify the presence of a rear facing child seat in the rear seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to or from the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. Nos. 5,829,782, and 5,943,295 among others, will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The instant invention is concerned with the process of adapting the vehicle interior monitoring systems to a particular vehicle model and achieving a high system accuracy and reliability as discussed in greater detail below as well as the resulting pattern recognition system architecture.
The automatic adjustment of the deployment rate of the airbag based on occupant identification and position and on crash severity has been termed xe2x80x9csmart airbagsxe2x80x9d. Central to the development of smart airbags is the occupant identification and position determination systems described in the above-referenced patents and patent applications and to the methods described herein for adapting those systems to a particular vehicle model. To complete the development of smart airbags, an anticipatory crash detecting system such as disclosed in U.S. patent application Ser. No. 08/247,760 filed May 23, 1994 is also desirable. Prior to the implementation of anticipatory crash sensing, the use of a neural network smart crash sensor which identifies the type of crash and thus its severity based on the early part of the crash acceleration signature should be developed and thereafter implemented. U.S. Pat. No. 5,684,701 (Breed) describes a crash sensor based on neural networks. This crash sensor, as with all other crash sensors, determines whether or not the crash is of sufficient severity to require deployment of the airbag and, if so, initiates the deployment. A neural network based on a smart airbag crash sensor could also be designed to identify the crash and categorize it with regard to severity thus permitting the airbag deployment to be matched not only to the characteristics and position of the occupant but also the severity and timing of the crash itself as described in more detail in U.S. Pat. No. 5,943,295 referenced above.
The need for an occupant out-of-position sensor has also been observed by others and several methods have been described in certain U.S. patents for determining the position of an occupant of a motor vehicle. However, no patents have been found that describe the methods of adapting such sensors to a particular vehicle model to obtain high system accuracy or to a resulting architecture combination of pattern recognition algorithms. Each of these systems will be discussed below and have significant limitations.
In White et al. (U.S. Pat. No. 5,071,160), for example, a single acoustic sensor and detector is described and, as illustrated, is disadvantageously mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors. White et al. does not disclose where the such sensors would be mounted, other than on the instrument panel below the steering wheel, or how they would be combined to uniquely monitor particular locations in the passenger compartment and to identify the object(s) occupying those locations. The adaptation process to vehicles is not described nor is a combination of pattern recognition algorithms.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods for measuring the change in position of an occupant including ultrasonic, active or passive infrared radiation, microwave radar sensors, and an electric eye. The use of these sensors is to measure the change in position of an occupant during a crash and they use that information to assess the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in the above-referenced patents and/or patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use a combination of ultrasonic sensors/transmitters to identify the presence of a human occupant and then to find his/her location in the passenger compartment or any pattern recognition algorithm let alone a combination of such algorithms.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant in the passenger compartment relative to the occupant protection apparatus, such as an airbag, since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. disclose only lower mounting locations of their sensors which are mounted in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant""s hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry, or pattern recognition system, which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver""s arms are the closest objects to two of the sensors. The determination of the proper transducer mounting locations, aiming and field angles and pattern recognition system architectures for a particular vehicle model are not disclosed in either White et al. or Mattes et al. and are part of the vehicle model adaptation process described herein.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant""s hands, as in the case where he/she is adjusting the knob on the radio, and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was moving. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant""s head or chest. Since the sizes and driving positions of occupants are extremely varied, trained pattern recognition systems, such as neural networks and combinations thereof, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed. White et al. do not suggest the use of such neural networks.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to control and suppress deployment of an airbag if the occupant is out-of-position, or if a rear facing child seat is present. In fact, the closer that the occupant gets to the airbag, the faster the inflation rate of the airbag is according to the Fujita et al. patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat. This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant""s head.
It is important to note that in all cases in the above-cited prior art, except those assigned to the current assignee of the instant invention, no mention is made of the method of determining transducer location, deriving the algorithms or other system parameters that allow the system to accurately identify and locate an object in the vehicle. In contrast, in one implementation of the instant invention, the return ultrasonic echo pattern over several milliseconds corresponding to the entire portion of the passenger compartment volume of interest is analyzed from multiple transducers and sometimes combined with the output from other transducers, providing distance information to many points on the items occupying the passenger compartment.
Many of the teachings of this invention are based on pattern recognition technologies as taught in numerous textbooks and technical papers. Central to the diagnostic teachings of this invention are the manner in which the diagnostic module determines a normal pattern from an abnormal pattern and the manner in which it decides what data to use from the vast amount of data available. This is accomplished using pattern recognition technologies, such as artificial neural networks, and training. The theory of neural networks including many examples can be found in several books on the subject including: Techniques And Application Of Neural Networks, edited by Taylor, M. and Lisboa, P., Ellis Horwood, West Sussex, England, 1993; Naturally Intelligent Systems, by Caudill, M. and Butler, C., MIT Press, Cambridge Mass., 1990; J. M. Zaruda, Introduction to Artificial Neural Systems, West publishing Co., N.Y., 1992 and, Digital Neural Networks, by Kung, S. Y., PTR Prentice Hall, Englewood Cliffs, N.J., 1993, Eberhart, R., Simpson, P. and Dobbins, R., Computational Intelligence PC Tools, Academic Press, Inc., 1996, Orlando, Fla., all of which are included herein by reference. The neural network pattern recognition technology is one of the most developed of pattern recognition technologies. The invention described herein uses combinations of neural networks to improve the pattern recognition process.
Other patents describing occupant sensor systems include U.S. Pat. No. 5,482,314 (Corrado et al.) and U.S. Pat. No. 5,890,085 (Corrado et al.). These patents describe a system for sensing the presence, position and type of an occupant in a seat of a vehicle for use in enabling or disabling a related airbag activator. A preferred implementation of the system includes two or more different but collocated sensors which provide information about the occupant and this information is fused or combined in a microprocessor circuit to produce an output signal to the airbag controller. According to Corrado et al., the fusion process produces a decision as to whether to enable or disable the airbag with a higher reliability than a single phenomena sensor or non-fused multiple sensors. By fusing the information from the sensors to make a determination as to the deployment of the airbag, each sensor has only a partial effect on the ultimate deployment determination. The sensor fusion process is a crude pattern recognition process based on deriving the fusion xe2x80x9crulesxe2x80x9d by a trial and error process rather than by training.
The sensor fusion method of Corrado et al. requires that information from the sensors be combined prior to processing by an algorithm in the microprocessor. This combination could be found to unnecessarily complicate the processing of the data from the sensors and other data processing methods might provide better results. For example, as discussed more fully below, it has been found to be advantageous to use a more efficient pattern recognition algorithm such as a combination of neural networks or fuzzy logic algorithms which are arranged to receive a separate stream of data from each sensor, without that data being combined with data from the other sensors (as in done in Corrado et al.) prior to analysis by the pattern recognition algorithms. In this regard, it is critical to appreciate that sensor fusion is a form of pattern recognition but is not a neural network and that significant and fundamental differences exist between sensor fusion and neural networks. Thus, some embodiments of the invention described below differ from that of Corrado et al. because they include a microprocessor which is arranged to accept only a separate stream of data from each sensor such that the stream of data from the sensors are not combined with one another. Further, the microprocessor processes each separate stream of data independent of the processing of the other streams of data (i.e., without the use of any fusion matrix as in Corrado et al.).
2. Definitions
The use of pattern recognition, or more particularly how it is used, is central to the instant invention. In the above-cited prior art, except in that assigned to the current assignee of the instant invention, pattern recognition which is based on training, as exemplified through the use of neural networks, is not mentioned for use in monitoring the interior passenger compartment or exterior environments of the vehicle. Thus, the methods used to adapt such systems to a vehicle are also not mentioned.
xe2x80x9cPattern recognitionxe2x80x9d as used herein will generally mean any system which processes a signal that is generated by an object (e.g., representative of a pattern of returned or received impulses, waves or other physical property specific to and/or characteristic of and/or representative of that object) or is modified by interacting with an object, in order to determine to which one of a set of classes that the object belongs. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally a series of electrical signals coming from transducers that are sensitive to acoustic (ultrasonic) or electromagnetic radiation (e.g., visible light, infrared radiation, capacitance or electric and magnetic fields), although other sources of information are frequently included. Pattern recognition systems generally involve the creation of a set of rules that permit the pattern to be recognized. These rules can be created by fuzzy logic systems, statistical correlations, or through sensor fusion methodologies as well as by trained pattern recognition systems such as neural networks.
A trainable or a trained pattern recognition system as used herein generally means a pattern recognition system which is taught to recognize various patterns constituted within the signals by subjecting the system to a variety of examples. The most successful such system is the neural network used either singly or as a combination of neural networks. Thus, to generate the pattern recognition algorithm, test data is first obtained which constitutes a plurality of sets of returned waves, or wave patterns, from an object (or from the space in which the object will be situated in the passenger compartment, i.e., the space above the seat) and an indication of the identify of that object. A number of different objects are tested to obtain the unique wave patterns from each object. As such, the algorithm is generated, and stored in a computer processor, and which can later be applied to provide the identity of an object based on the wave pattern being received during use by a receiver connected to the processor and other information. For the purposes here, the identity of an object sometimes applies to not only the object itself but also to its location and/or orientation in the passenger compartment. For example, a rear facing child seat is a different object than a forward facing child seat and an out-of-position adult is a different object than a normally seated adult.
To xe2x80x9cidentityxe2x80x9d as used herein will generally mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
An xe2x80x9cobjectxe2x80x9d in a vehicle or an xe2x80x9coccupying itemxe2x80x9d of a seat may be a living occupant such as a human or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries or an empty child seat.
xe2x80x9cOut-of-positionxe2x80x9d as used for an occupant will generally mean that the occupant, either the driver or a passenger, is sufficiently close to the occupant protection apparatus (airbag) prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. It may also mean that the occupant is not positioned appropriately in order to attain beneficial, restraining effects of the deployment of the airbag. An occupant is too close to the airbag when the occupant""s head or chest is closer than some distance, such as about 5 inches, from the deployment door of the airbag module. The actual distance value where airbag deployment should be suppressed depends on the design of the airbag module and is typically farther for the passenger airbag than for the driver airbag.
xe2x80x9cTransducerxe2x80x9d as used herein will generally mean the combination of a transmitter and a receiver. In come cases, the same device will serve both as the transmitter and receiver while in others two separate devices adjacent to each other will be used. In some cases, a transmitter is not used and in such cases transducer will mean only a receiver. Transducers include, for example, capacitive, inductive, ultrasonic, electromagnetic (antenna, CCD, CMOS arrays), weight measuring or sensing devices.
xe2x80x9cAdaptationxe2x80x9d as used here will generally represent the method by which a particular occupant sensing system is designed and arranged for a particular vehicle model. It includes such things as the process by which the number, kind and location of various transducers is determined. For pattern recognition systems, it includes the process by which the pattern recognition system is designed and then taught to recognize the desired patterns. In this connection, it will usually include (1) the method of training, (2) the makeup of the databases used for training, testing and validating the particular system, or, in the case of a neural network, the particular network architecture chosen, (3) the process by which environmental influences are incorporated into the system, and (4) any process for determining the pre-processing of the data or the post processing of the results of the pattern recognition system. The above list is illustrative and not exhaustive. Basically, adaptation includes all of the steps that are undertaken to adapt transducers and other sources of information to a particular vehicle to create the system which accurately identifies and determines the location of an occupant or other object in a vehicle.
A xe2x80x9ccombination neural networkxe2x80x9d as used herein will generally apply to any combination of two or more neural networks that are either connected together or that analyze all or a portion of the input data. A combination neural network can be used to divide tasks in solving a particular occupant problem. For example, one neural network can be used to identify an object occupying a passenger compartment of an automobile and a second neural network can be used to determine the position of the object or its location with respect to the airbag, for example, within the passenger compartment. In another case, one neural network can be used merely to determine whether the data is similar to data upon which a main neural network has trained or whether there is something radically different about this data and therefore that the data should not be analyzed.
With respect to a comparative analysis performed by neural networks to that perform by the human mind, once the human mind has identified that the object observer is a tree, the mind does not try to determine whether it is the black bear or a grizzly. Further observation on the tree might center on whether it is a pine tree, an oak tree etc. Thus the human mind appears to operate in some manner like a hierarchy of neural networks. Similarly, neural networks for analyzing the occupancy of the vehicle can be structured such that higher order networks are used to determine, for example, whether there is an occupying item of any kind present. This could be followed by the neural network that, knowing that there is information on the item, attempts to categorize the item into child seats and human adults etc., i.e., determine the type of item Once it has decided that a child seat is present, then another neural network can be used to determine whether the child seat is rear facing or forward facing. Once the decision has been made that the child seat is facing rearward, the position of the child seat relative to the airbag, for example, can be handled by still another neural network. The overall accuracy of the system can be substantially improved by breaking the pattern recognition process down into a larger number of smaller pattern recognition problems.
In some cases, the accuracy of the pattern recognition process can be improved if the neural network has data indicating its own recent decisions. Thus, for example, if the neural network system had determined that a forward facing adult was present, then that information can be used as input into another neural network, biasing any results toward the forward facing human compared to a rear facing child seat, for example. Similarly, for the case when an occupant is being tracked in his or her forward motion during a crash, for example, the location of the occupant at the previous calculation time step can be valuable information to determining the location of the occupant from the current data. There is a limited distance an occupant can move in 10 milliseconds, for example. In this latter example, feedback of the decision of the neural network tracking algorithm becomes important input into the same algorithm for the calculation of the position of the occupant at the next time step.
What has been described above is generally referred to as modular neural networks with and without feedback. Actually, the feedback does not have to be from the output to the input of the same neural network. The feedback from a downstream neural network could be input to an upstream neural network, for example.
The neural networks can be combined in other ways, for example in a voting situation. Sometimes the data upon which the system is trained is sufficiently complex or imprecise that different views of the data will give different results. For example, a subset of transducers may be used to train one neural network and another subset to train a second neural network etc. The decision can then be based on a voting of the parallel neural networks, known as an ensemble neural networks. In the past, neural networks have usually only been used in the form of a single neural network algorithm for identifying the occupancy state of an automobile. This invention is primarily advancing the state of the art and using combination neural networks wherein two or more neural networks are combined to arrive at a decision.
In the description herein on anticipatory sensing, the term xe2x80x9capproachingxe2x80x9d when used in connection with the mention of an object or vehicle approaching another will generally mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The xe2x80x9ctargetxe2x80x9d vehicle is the vehicle which is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
Also, for the purposes herein, a xe2x80x9cwave sensorxe2x80x9d or xe2x80x9cwave transducerxe2x80x9d is generally any device, which senses either ultrasonic or electromagnetic waves . An electromagnetic wave sensor, for example, includes devices that sense any portion of the electromagnetic spectrum from ultraviolet down to a few hertz. The most commonly used kinds of electromagnetic wave sensors include CCD and CMOS arrays for sensing visible and/or infrared waves, millimeter wave and microwave radar, and capacitive or electric and magnetic field monitoring sensors that rely on the dielectric constant of the object occupying a space but also rely on the time variation of the field, expressed by waves, to determine a change in state. In this regard, reference is made to, for example, U.S. patents by Kithil et al. U.S. Pat. Nos. 5,366,241, 5,602,734, 5,691,693, 5,802,479 and 5,844,486 and Jinno et al. U.S. Pat. No. 5,948,031 which are included herein by reference.
3. Pattern Recognition Prior Art
Japanese Patent No. 3-42337 (A) to Ueno describes a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used. The method of finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver""s mouth and eyes. As with Ueno, a method of finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,298,732 to Chen also concentrates on locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver""s eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of an automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, in Chen, there is no mention of monitoring the position of the occupant, other that the eyes, determining the position of the eyes relative to the passenger compartment, or identifying any other object in the vehicle other than the driver""s eyes. Also, there is no mention of the use of a trainable pattern recognition system. A method for finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant by using two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Faris is only interested in locating the driver""s eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant, a rear facing child seat or any other object for that matter, relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with ultrasonic or electromagnetic radiation as in some implementations of the instant invention. A method for finding the eyes of the occupant is described but not a method of adapting the system to a particular vehicle model.
The use of neural networks, or neural fuzzy systems, and particular combination neural networks, as the pattern recognition technology and the methods of adapting this to a particular vehicle, such as the training methods, is important to this invention since it makes the monitoring system robust, reliable and accurate. The resulting systems are easy to implement at a low cost making them practical for automotive applications. The cost of the ultrasonic transducers, for example, is expected to be less than about $1 in quantities of one million per year and CMOS cameras, currently less than $5 each in similar quantities. Similarly, the implementation of the techniques of the above-referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than about $5 in quantities of one million per year.
The present invention uses sophisticated software that develops trainable pattern recognition algorithms such as neural networks and combination neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques and the results post-processed to improve system accuracy. A non-automotive example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. xe2x80x9cAnalysis of Hidden Units in a Layered Network Trained to Classify Sonar Targetsxe2x80x9d, Neural Networks, Vol. 1. pp. 75-89, 1988, and xe2x80x9cLearned Classification of Sonar Targets Using a Massively Parallel Networkxe2x80x9d, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled xe2x80x9cProcess and Apparatus for the Automatic Detection and Extraction of Features in Images and Displaysxe2x80x9d to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled xe2x80x9cArtificial Neuron and Method of Using Samexe2x80x9d and U.S. Pat. No. 5,517,667 entitled xe2x80x9cNeural Network That Does Not Require Repetitive Trainingxe2x80x9d to Wang, S. T.. Other examples include U.S. Pat. No. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744 (Schweizer et al), 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). All of the references herein are included herein by reference.
4. Ultrasonics and Optics
Both laser and non-laser optical systems in general are good at determining the location of objects within the two dimensional plane of the image and a pulsed laser radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time of flight through range gating techniques. Distance can also be determined by using modulated electromagnetic radiation and measuring the phase difference between the transmitted and received waves. It is also possible to determine distance with the non-laser system by focusing, or stereographically if two spaced apart receivers are used and, in some cases, the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements as discussed in U.S. provisional patent application Ser. No. 60/114,507, filed Dec. 31, 1998, which is included herein by reference as if the entire contents were copied here.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z-axis is from the transducer to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD or CMOS arrays are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. Nos. 5,845,000 and 5,835,613 cross-referenced above.
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and a receiver array, e.g., CCD or CMOS array, which receives and displays the reflected light and an analog to digital converter (ADC) which digitizes the output of the CCD or CMOS and feeds it to an Artificial Neural Network (ANN) or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the driver""s ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant""s ears.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the occupant can be updated to approximately 10 milliseconds, which though sufficient for most cases, is marginal if the position of the occupant is to be tracked during a vehicle crash. Second, ultrasound waves are diffracted by changes in air density that can occur when the heater or air conditioner is operated or when there is a high-speed flow of air past the transducer. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
Ultrasonics alone can be used in several configurations for monitoring the interior of a passenger compartment of an automobile as described in the above-referenced patents and patent applications and in particular in U.S. Pat. No. 5,943,295. Using the teachings of this invention, the optimum number and location of the ultrasonic and/or optical transducers can be determined as part of the adaptation process for a particular vehicle model.
In the cases of the instant invention, as discussed in more detail below, regardless of the number of transducers used, a trained pattern recognition system, as defined above, is used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
5. Applications
The applications for this technology are numerous as described in the patents and patent applications listed above. However, the main focus of the instant invention is the process and resulting apparatus of adapting the system in the patents and patent applications referenced above and using combination neural networks for the detection of the presence of an occupied child seat in the rear facing position or an out-of-position occupant and the detection of an occupant in a normal seating position. The system is designed so that in the former two cases, deployment of the occupant protection apparatus (airbag) may be controlled and possibly suppressed, and in the latter case, it will be controlled and enabled.
One preferred implementation of a first generation occupant sensing system, which is adapted to various vehicle models using the teachings presented herein, is an ultrasonic occupant position sensor. This system uses a Combination Artificial Neural Network (CANN) to recognize patterns that it has been trained to identify as either airbag enable or airbag disable conditions. The pattern is obtained from four ultrasonic transducers that cover the front passenger seating area. This pattern consists of the ultrasonic echoes bouncing off of the objects in the passenger seat area. The signal from each of the four transducers consists of the electrical image of the return echoes, which is processed by the electronics. The electronic processing comprises amplification, logarithmic compression rectification, and demodulation (band pass filtering), followed by discretization (sampling) and digitization of the signal. The only software processing required, before this signal can be fed into the combination artificial neural network, is normalization (i.e., mapping the input to numbers between 0 and 1). Although this is a fair amount of processing, the resulting signal is still considered xe2x80x9crawxe2x80x9d, because all information is treated equally.
In general, it is an object of the present invention to provide a new and improved system for identifying the presence, position and/or orientation of an object in a vehicle.
It is another broad object of the present invention to provide a system for accurately detecting the presence of an occupied rear-facing child seat in order to prevent an occupant protection apparatus, such as an airbag, from deploying, when the airbag would impact against the rear-facing child seat if deployed.
It is yet another broad object of the present invention to provide a system for accurately detecting the presence of an out-of-position occupant in order to prevent one or more deployable occupant protection apparatus such as airbags from deploying when the airbag(s) would impact against the head or chest of the occupant during its initial deployment phase causing injury or possible death to the occupant.
This invention is a system designed to identify, locate and monitor occupants, including their parts, and other objects in the passenger compartment and in particular an occupied child seat in the rear facing position or an out-of-position occupant, by illuminating the contents of the vehicle with ultrasonic or electromagnetic radiation, for example, by transmitting radiation waves from a wave generating apparatus into a space above the seat, and receiving radiation modified by passing through the space above the seat using two or more transducers properly located in the vehicle passenger compartment, in specific predetermined optimum locations. More particularly, this invention relates to a system including a plurality of transducers appropriately located and mounted and which analyze the received radiation from any object which modifies the waves, or which analyze a change in the received radiation caused by the presence of the object (e.g., a change in the dielectric constant), in order to achieve an accuracy of recognition heretofore not possible. Outputs from the receivers are analyzed by appropriate computational means employing trained pattern recognition technologies, and in particular combination neural networks, to classify, identify and/or locate the contents, and/or determine the orientation of, for example, a rear facing child seat. In general, the information obtained by the identification and monitoring system is used to affect the operation of some other system, component or device in the vehicle and particularly the passenger and/or driver airbag systems , which may include a front airbag, a side airbag, a knee bolster, or combinations of the same. However, the information obtained can be used for controlling or affecting the operation of a multitude of other vehicle systems.
When the vehicle interior monitoring system in accordance with the invention is installed in the passenger compartment of an automotive vehicle equipped with a occupant protection apparatus, such as an inflatable airbag, and the vehicle is subjected to a crash of sufficient severity that the crash sensor has determined that the protection apparatus is to be deployed, the system has determined (usually prior to the deployment) whether a child placed in the child seat in the rear facing position is present and if so, a signal has been sent to the control circuitry that the airbag should be controlled and most likely disabled and not deployed in the crash. It must be understood though that instead of suppressing deployment, it is possible that the deployment may be controlled so that it might provide some meaningful protection for the occupied rear-facing child seat. The system developed using the teachings of this invention also determines the position of the vehicle occupant relative to the airbag and controls and possibly disables deployment of the airbag if the occupant is positioned so that he/she is likely to be injured by the deployment of the airbag. As before, the deployment is not necessarily disabled but may be controlled to provide protection for the out-of-position occupant.
Principle objects and advantages of the methods in accordance with the invention are:
1. To provide a reliable system for recognizing the presence of a rear-facing child seat on a particular seat of a motor vehicle.
2. To provide a reliable system for recognizing the presence of a human being on a particular seat of a motor vehicle.
3. To provide a reliable system for determining the position, velocity or size of an occupant in a motor vehicle.
4. To provide a reliable system for determining in a timely manner that an occupant is out-of-position, or will become out-of-position, and likely to be injured by a deploying airbag.
5. To provide a system in which transducers are located within the passenger compartment at specific locations such that a high reliability of classification of objects and their position is obtained from the signals generated by the transducers.
6. To provide a system including a variety of transducers such as seatbelt payout sensors, seatbelt buckle sensors, seat position sensors, seatback position sensors, and weight sensors and which is adapted so as to constitute a highly reliable occupant presence and position system when used in combination with electromagnetic, ultrasonic or other radiation sensors.
Accordingly, in a vehicle including system for determining the occupancy state of a seat therein in accordance with the invention, the system comprises a plurality of transducers arranged in the vehicle, each transducers providing data relating to the occupancy state of the seat, and processor means or a processing unit (e.g., a microprocessor) coupled to the transducers for receiving the data from the transducers and processing the data to obtain an output indicative of the current occupancy state of the seat. The processor means comprise a combination neural network algorithm created from a plurality of data sets, each representing a different occupancy state of the seat and being formed from data from the transducers while the seat is in that occupancy state. The combination neural network algorithm produces the output indicative of the current occupancy state of the seat upon inputting a data set representing the current occupancy state of the seat and being formed from data from the transducers. The algorithm may be a pattern recognition algorithm or neural network algorithm generated by a combination neural network algorithm-generating program.
In accordance with some embodiments of the invention, the processor means are arranged to accept only a separate stream of data from each transducer such that the stream of data from each transducer is passed to the processor means without combining with another stream of data. Further, the processor means may be arranged to process each separate stream of data independent of the processing of the other streams of data.
It is an important feature of the invention that the transducers may be selected from a wide variety of different sensors, all of which are affected by the occupancy state of the seat. That is, different combinations of known sensors can be utilized in the many variations of the invention. For example, the sensors used in the invention may include a weight sensor arranged in the seat, a reclining angle detecting sensor for detecting a tilt angle of the seat between a back portion of the seat and a seat portion of the seat, a seat position sensor for detecting the position of the seat relative to a fixed reference point in the vehicle, a heartbeat sensor for sensing a heartbeat of an occupying item of the seat, a capacitive sensor, a seat belt buckle sensor, a seatbelt payout sensor, an infrared sensors, an inductive sensor, a motion sensor and a radar sensor. The same type of sensor could also be used, preferably situated in a different location, but possibly in the same location for redundancy purposes. For example, the system may include a plurality of weight sensors, each measuring the weight applied onto the seat at a different location. Such weight sensors may include a weight sensor, such as a strain gage, arranged to measure displacement of a surface of a seat portion of the seat and/or a strain, force or pressure gage arranged to measure displacement of the entire seat. In the latter case, the seat includes a support structure for supporting the seat above a floor of a passenger compartment of the vehicle whereby the strain gage can be attached to the support structure.
In some embodiments, the transducers include a plurality of electromagnetic wave sensors capable of receiving waves at least from a space above the seat, each electromagnetic wave sensor being arranged at a different location. Other wave sensors, such as capacitive sensors, can also be used.
In other embodiments, the transducers include at least two ultrasonic sensors capable of receiving waves at least from a space above the seat, each ultrasonic sensor being arranged at a different location. For example, one sensor is arranged on a ceiling of the vehicle and the other is arranged at a different location in the vehicle, preferably so that an axis connecting the sensors is substantially parallel to a second axis traversing a volume in the vehicle above the seat. The second sensor may be arranged on a dashboard or instrument panel of the vehicle. A third ultrasonic sensor can be arranged on an interior side surface of the passenger compartment while a fourth can be arranged on or adjacent an interior side surface of the passenger compartment. The ultrasonic sensors are capable of transmitting waves at least into the space above the seat. Further, the ultrasonic sensors are preferably aimed such that the ultrasonic fields generated thereby cover a substantial portion of the volume surrounding the seat. Horns or grills may be provided for adjusting the transducer field angles of the ultrasonic sensors to reduce reflections off of fixed surfaces within the vehicle or otherwise control the shape of the ultrasonic field.
The actual location of the ultrasonic sensors can be determined by placing a significant number of ultrasonic sensors in the vehicle and removing those sensors which prove analytically to be redundant.
The ultrasonic sensors can have different transmitting and receiving frequencies and be arranged in the vehicle such that sensors having adjacent transmitting and receiving frequencies are not within a direct ultrasonic field of each other.
Once the occupancy state of the seat (or seats) in the vehicle is known, this information can be used to control or affect the operation of a significant number of vehicular systems, components and devices. That is, the systems, components and devices in the vehicle would be controlled in consideration of the occupancy of the seat(s) in the vehicle, possibly to optimize operation of the same. Thus, the vehicle includes control means coupled to the processor means for controlling a component or device in the vehicle in consideration of the output indicative of the current occupancy state of the seat obtained from the processor means. The component or device can be an airbag system including at least one deployable airbag whereby the deployment of the airbag is suppressed, e.g., if the seat is occupied by a rear-facing child seat, or otherwise the parameters of the deployment are controlled.
In another embodiment of the invention, the system for determining the occupancy state of a seat in a vehicle includes a plurality of transducers arranged in the vehicle, each providing data relating to the occupancy state of the seat, and processor means coupled to the transducers for receiving only a separate stream of data from each transducer (such that the stream of data from each transducer is passed to the processor means without combining with another stream of data) and processing the streams of data to obtain an output indicative of the current occupancy state of the seat. The processor means comprise an algorithm created from a plurality of data sets, each representing a different occupancy state of the seat and being formed from separate streams of data, each only from one transducers, while the seat is in that occupancy state. The algorithm produces the output indicative of the current occupancy state of the seat upon inputting a data set representing the current occupancy state of the seat and being formed from separate streams of data, each only from one transducer. The processor means preferably process each separate stream of data independent of the processing of the other streams of data.
In yet another embodiment of the invention, the system for determining the occupancy state of a seat in a vehicle includes a plurality of transducers including at least two wave-receiving transducers arranged in the vehicle, each providing data relating to the occupancy state of the seat. One wave-receiving transducer is arranged on or adjacent to a ceiling of the vehicle and a second wave-receiving transducer is arranged at a different location in the vehicle such that an axis connecting these wave-receiving transducers is substantially parallel to a longitudinal axis of the vehicle, substantially parallel to a transverse axis of the vehicle or passes through a volume above the seat. A processor is coupled to the transducers for receiving data from the transducers and processing the data to obtain an output indicative of the current occupancy state of the seat. The processor comprises an algorithm which produces the output indicative of the current occupancy state of the seat upon inputting a data set representing the current occupancy state of the seat and being formed from data from the transducers.
In still another embodiment of the invention, the system includes a plurality of transducers arranged in the vehicle, each providing data relating to the occupancy state of the seat, and which include a wave-receiving transducer and a non-wave-receiving transducer. The system also includes processor means coupled to the transducers for receiving the data from the transducers and processing the data to obtain an output indicative of the current occupancy state of the seat. The processor means comprising an algorithm created from a plurality of data sets, each representing a different occupancy state of the seat and being formed from data from the transducers while the seat is in that occupancy state. The algorithm produces the output indicative of the current occupancy state of the seat upon inputting a data set representing the current occupancy state of the seat and being formed from data from the transducers.
In all of the implementations of the invention described above, a combination or combinational neural network is used. The particular combination neural network is determined by a process in which a number of neural network modules are combined in a parallel and a serial manner and an optimization program can be utilized to determine the best combination of such neural networks to achieve the highest accuracy. Alternately, the optimization process can be undertaken manually in a trial and error manner. In this manner, the optimum combination of neural networks is selected to solve the particular pattern recognition and categorization objective desired.
These and other objects and advantages will become apparent from the following description of the preferred embodiments of the vehicle identification and monitoring system of this invention.